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FOREWORD 


The  National  Training  Center  (NTC)  is  recognized  as  the  closest  to  a  real 
combat  environment  currently  available  to  the  Army,  but  the  expense  of  train¬ 
ing  at  the  NTC  precludes  frequent  rotations  by  any  single  unit.  Other  means 
must  be  found  that  provide  greater  cost  efficiencies  and  effectiveness  for 
unit  training  at  home  station.  Simulation  Networking  (SIMNET)  may  provide 
such  an  alternative. 

Strategies  are  needed  for  Integrating  SIMNET  training  into  the  Army 
training  system.  This  report  addresses  a  needed  element  in  the  development  of 
such  strategies.  It  presents  a  concept  for  the  design  of  a  common  performance 
measurement  system  for  unit  training  in  the  NTC  and  SIMNET  environments.  This 
concept  is  being  applied  in  an  ARI  research  effort  to  develop  SIMNET  training 
methods  and  strategies. 


EDGAR  M.  JOHNSON 
Technical  Director 


CONCEPT  FOR  A  COMMON  PERFORMANCE  MEASUREMENT  SYSTEM  FOR  UNIT  TRAINING  AT  THE 
NATIONAL  TRAINING  CENTER  (NTC)  AND  WITH  SIMULATION  NETWORKING  (SIMNET) 


EXECUTIVE  SUMMARY 


Requirement: 

The  research  described  in  this  paper  was  part  of  a  larger  effort 
sponsored  by  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI)  with  participation  by  the  Combined  Arms  Training  Activity 
(CATA).  The  larger  effort  had  three  primary  objectives:  (1)  to  design  a 
performance  measurement  concept  for  SIMNET  technology  based  on  previous  ARI/ 
CATA  sponsored  research  focused  on  improving  the  Army  Training  and  Evaluation 
Program  (ARTEP)  using  the  NTC  as  the  developmental  vehicle;  (2)  to  design  and 
evaluate  a  training  management  system  for  SIMNET  technology  (e.g.,  identifica¬ 
tion  of  prerequisite  skills,  design  of  training  scenarios,  strategies  for 
training  feedback);  and  (3)  to  design  training  strategies  for  incorporating 
SIMNET  training  into  the  Army  training  system. 

This  research  focused  on  the  first  objective  and  presents  a  concept  for 
the  design  of  a  common  performance  measurement  system  for  unit  training  at  the 
National  Training  Center  (NTC)  and  Simulation  Networking  (SIMNET).  Two  pla¬ 
toon  missions  (defense  and  hasty  attack)  were  used  as  the  vehicle  for  present¬ 
ing  the  concept. 


Procedure: 

The  measurement  model  established  in  previous  ARI  research  on  unit  per¬ 
formance  measurement  and  used  to  guide  the  present  effort  is  a  three-part 
model  that  recognizes  the  roles  of  process,  outcome,  and  expert  judgment  in 
assessing  unit  performance.  The  model  provides  a  "bottom  line"  outcome 
measure  of  mission,  enemy  forces,  friendly  troops,  terrain,  and  time  (METT-T). 
In  addition  to  METT-T  factors,  ARTEP  Mission-Training-Plan- (MTP)  based 
mission-critical  tasks  have  been  Identified  that  provide  a  benchmark  for 
measuring  the  degree  of  unit  conformity  to  established  tactical  doctrine. 

The  concept  for  a  common  performance  measurement  system  (across  both  NTC 
and  SIMNET  training)  described  herein  examines  the  applicability  of  the  NTC- 
based  model  to  the  SIMNET  environment  and  identifies  the  boundaries  of  overlap 
between  the  two  training  settings. 


Findings: 

Three  types  of  measures  of  performance  (MOPs)  were  generated  for  all  NTC- 
based  critical  tasks  identified  as  supportable  or  potentially  supportable  in 


SIMNET  based  on  the  specific  requirements  of  each  critical  task:  (1)  auto¬ 
mated  MOPs  ^ those  that  can  be  collected  from  the  SIMNET  data  stream; 

(2)  observable  MOPs  (i.e.,  those  that  can  be  gathered  by  direct  observation 
of  task  performance,  indirect  observation  using  the  plan  view  display,  and/or 
expert  judgment  based  on  syntheses  of  observable  and  automated  MOPs;  auid 

(3)  "other"  MOPs  (those  that  would  require  a  modification  either  in  the  SIMNET 
facility--such  as  the  acquisition  of  a  multi-channel  recorder  for  radio  nets-- 
or  in  the  SIMNET  simulation  software). 


Utilization  of  Findings: 

The  measurement  concept,  to  include  METT-T  factors,  identified  critical 
tasks.  MOPs  now  must  be  further  elaborated  through  a  systematic  design  pro¬ 
cess  to  build  a  fully  functional  feedback  system.  This  process  Includes  the 
design  of  data  collection  tools  (including  screen  displays  for  automated  data 
collection),  data  analysis  plans,  structured  approaches  for  data  interpreta¬ 
tion,  and  strategies  for  feedback  presentation.  These  processes  were 
addressed  in  the  other  requirements  of  the  larger  research  effort. 
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COf’CEPT  FOP  .*  COMMON  PERFORMANCE  MEASUREMENT  SYSTEM  FOR  UNIT 
TRAINING  iHE  NATIONAL  TRAINING  CENTER  (NTC)  AND  WITH  SIMULATION 

NETWORKING  (SIMNET) 


Purpose 

The  purpose  of  this  report  is  to  present  a  concept  for  the  design  of  a  comnwn  performance 
measurement  system  for  unit  training  at  the  National  Training  Center  (NTC)  and  with  Simulation 
Networking  (SIMNET).  In  this  report,  two  platoon  missions  (Defense,  Hasty  Attack)  arc  used  as 
the  vehicle  for  presenting  the  concept.  The  methodology  has  bwn  applied  to  the  Platoon  and 
Company  echelons  for  the  Defense,  Hasty  Attack,  and  Movc.ment  to  Contact  missions.  The  results 
of  these  applications  are  contained  in  separate  publications  (Kerins  and  Atwood,  1990a, b,c; 
Kerins,  Root,  and  Atwood  1989a,b  c).  These  missions  and  echelons  were  identified  based  on 
guidance  about  frequency  and  tj-pe  of  training  missions  anticipated  in  initial  SIMNET  usage 
for  unit  training. 

The  research  described  here  is  part  of  a  larger  project  sponsored  by  the  U.S.  Army 
Research  Institute  (ARI)  with  participation  by  the  Combined  Arms  Training  Activity  ((2ATA).  The 
project  has  three  primary  objectives:  1)  to  design  a  performance  measurement  concept  for  the 
SIMNET  technology  keying  off  previous  ARI/CATA  sponsored  research  on  an  improved  Army 
Training  and  Evaluation  Program  (ARTEP)  that  has  focused  on  the  NTC  for  development:  2)  to 
design  and  evaluate  a  training  management  system  for  the  SIMNET  technology  (for  example, 
identification  of  prerequisite  skills,  design  of  training  scenarios,  strategies  for  training  feedback); 
and  3)  to  design  training  strategies  for  incorporating  SIMNET  training  into  unit  training  programs 
(along  with  other  types  of  training  such  as  UCOIT,  ARTBASS,  FTXs,  CPXs  among  others). 

This  research  repon  is  the  initial  project  deliverable  focused  on  the  first  objective  of  the 
project.  It  is  organized  into  five  major  sections:  background,  measurement  approach, 
methodology  for  application  to  SIMNET,  key  findings,  and  next  steps. 

Background 

Simulation  Networking  (SIMNET)  is  an  innovative  simulation  that  uses  computer  driven 
networked  training  devices  to  support  combined  arms  training  at  unit  home  stations.  SIMNET 
is  a  projea  of  the  Defense  Advanc^  Research  Project  Agency  (DARPA)  with  Perceptronics  as  the 
prime  design  and  development  contractor.  Currently,  SIMNCT  technolog)’  is  being  used  to 
provide  fully  interactive  low-cost  simulators  that  permit  tactical  training  of  heavy  units  in  a  simu¬ 
lated  combined  arms  combat  environment.  These  full-crew  modules  can  operate  independently  or 
can  be  networked  locally  (and  eventually  over  long  distances  with  a  long  haul  network)  to  form 
platoons,  companies,  or  battalions. 

With  the  Army  decision  to  field  SIMNET  at  unit  home  stations  and  to  s^ure  production 
versions  with  even  greater  capabilities  (the  Qosc  (jombat  Tactical  Trainer  or  CCTT),  the  Army 
training  community  faces  a  host  of  important  questions  on  what  the  role  of  SIMNET  should  be  in  a 
unit  training  program,  how  SIMNET  training  should  be  conducted,  and  on  what  basis  unit  perfor¬ 
mance  should  be  measured.  These  are  complex  questions  with  important  policy  and  technical 
considerations. 

The  Training  Research  Laboratory  of  the  Army  Research  Institute  is  charged  with  the 
mission  of  conducting  research  on  emerging  and  systemic  issues  on  training  the  force  and 
providing  senior  Anny  leadership  with  research-based  recommendations  on  Army  training.  ARI 
has  undertaken  this  exploratory  research  effort  to  examine  critical  issues  emerging  on  SIMNET 
training.  The  pcrform.ance  measurement  asp)ect  of  SIMNET  collective  training  is  a  natural 
extension  of  previous  ARI  research  on  unit  performance  measurement 
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This  previous  ARI  research  on  unit  performance  measurement  has  been  conducted  at  the 
ARI  Field  Unit  at  the  Presidio  of  Monterey  with  contract  support  from  The  BDM  Corporation. 

The  current  p^ormance  measurement  extension  to  SIMNET  is  being  performed  by  BDM  for  the 
ARI  Field  Unit  at  Fort  Knox  under  a  Perceptronics,  Inc.  contract  with  Los  Alamos  National 
Laboratory.  The  research  is  predicated  on  the  Army's  overall  training  philosophy  as  laid  out  in  FM 
25-100,  the  Army's  capstone  training  manual.  At  the  heart  of  this  philosophy  is  the  concept  of 
train-evaluate-train. 

As  shown  in  Fi^re  1,  training  is  seen  as  an  iterative  process  which  requires  explicit 
statement  of  trying  objectives  and  requirements,  formulation  and  execution  of  a  training  strategy 
and  an  evaluation  of  performance.  Thus,  performance  assessment  is  a  key  feature  of  the  training 
management  cycle.  A  system  for  measuring  performance  is  required  to  assess  a  unit's  capability  in 
order  to  determine  training  requirements.  Such  a  system  is  also  required  to  assess  the  outcomes  of 
training  and  to  determine  needs  for  follow-on  training.  In  essence,  a  performance  assessment 
system  provides  the  diagnostic  feedback  that  ties  the  entire  training  management  cycle  together  and 
allows  leaders  to  assess  the  strengths  and  weaknesses  of  their  unit  (i.e.,  ^eir  training  status). 


WARTIME 

MISSION 

I 

ESTABLISH 
MISSION  ESSENTIAL 
TASK  LIST 


Figure  1.  Training  management  cycle  (from  FM  25-l(X)). 
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Tactical  doctrine  and  training  literamre  provide  basic  guidance  for  training  at  home  station 
generally  and  for  advanced  collective  training  at  the  Army's  Combat  Training  Centers  (CTCs). 
Training  guidance  literature  (Army  Training  and  Evaluation  Programs  [ARTEPs]  and  Army 
Mission  Training  Plans  [AM^s])  prescribe  tasks  that  allow  scoring  of  unit  performance  in  order 
to  determine  how  closely  performance  conforms  to  doctrine.  However,  ARTEPs/AMTPs  in  their 
current  form  are  impractical  to  use  for  assessing  unit  performance  particularly  in  advanced  col¬ 
lective  training  environments  such  as  the  CTCs  (see  Hiller,  1988).  More  specifically,  the  use  of 
evaluation  checklists  based  on  the  AMTP  have  considerable  practical  difficulties  when  they  arc 
implemented  as  scoring  protocols  for  field  exercises. 

Some  of  these  difficulties  arise  from  the  lack  of  priority  to  given  performance  elements, 
uncertain  relationships  to  MFTT-T  factors  for  these  elements,  and  the  lack  of  an  organizing 
structure.  Furthermore,  they  uo  not  take  into  account  the  unique  capabilities  of  computer-enhanced 
training  (such  as  the  instrumented  battlefield  of  the  NTC  or  computer-  driven  simulations  such  as 
SIMNET)  for  assessing  unit  performance.  Finally,  they  do  not  include  a  strategy  for  assessing 
overall  mission  effectiveness.  Such  objective  measures  of  performance  are  important  benchmarics 
for  gauging  the  validity  of  training  and  doctrine  and  for  identifying  lessons  learned  on  systemic 
improvements  in  these  areas. 

ARI  has  been  pursuing  a  research  program  to  develop  methodologies  to  strengthen  the 
ARTEP/.AMTP  programs  as  unit  performance  measurement  tools.  The  focus  of  this  research  has 
been  on  the  NT(i)  because  of  its  maturity  as  a  training  site,  the  realism  of  the  training  experience  for 
units  and  the  central  role  of  the  CTC  program  in  providing  advanced  collective  training.  Briefly, 
the  NTC  at  Fort  Irwin,  California  was  the  first  CTC  established  by  the  Army  and  has  been 
operational  for  over  seven  years.  The  NTC  provides  mission  essential  combined  arms  training  for 
armor  and  mechanized  infantry  forces.  The  NTC  has  several  critical  features  that  contribute  to  a 
realistic  combat  simulation:  1)  a  constrained  free-play  traitung  environment  governed  by  rules  of 
engagement;  2)  a  dedicated  Opposing  Force  which  uses  Soviet  tactics  and  equipment;  3)  a 
battlefield  environment  which  includes  electronic  warfare  and  close  air  support;  4)  extensive  use  of 
tactical  engagement  simulation  including  MILES  (Multiple  Integrated  Laser  Engagement 
Simulation);  and  5)  a  complex  instrumentation  system  for  real-time  data  collection  of  mission 
events.  Although  the  NTC  does  not  provide  a  complete  combat  simulation  due  to  deficiencies  in 
instrumentation  for  indirect  fire,  mobility  and  countermobility,  and  air  support  and  air  defense,  it 
does  stress  heavy  forces  in  the  use  oi  their  organic  weapon  systems. 

SIMNET  is  a  natural  counterpart  to  the  NTC  because  of  its  orientation  on  heavy  units  and 
its  capability  to  simulate  a  combined  arms  battlefield  under  free  play  conditions.  The  following 
section  describes  the  measurement  approach  developed  in  the  ARI  research  based  on  the  NTC. 
Subs^uent  sections  describe  the  methodology  used  to  apply  the  approach  to  SIMNET  and 
highlight  key  findings  from  this  application. 

Measurement  Approach 

The  measurement  model  used  to  guide  the  ARI  research  program  on  unit  performance 
measurement  is  shown  in  Figure  2.  The  ihrec-pan  model  recognizes  the  roles  of  prxxress, 
outcome  and  expert  judgment  in  assessing  unit  performance  and  specifies  three  forms  of 
measurement. 

The  mission  results  portion  of  the  nnnlel  provides  a  "bottom  line"  outcome  tneasine  r>l 
mission  aecomplishmcnl  (Root  and  Zimmerman,  in  preparation).  These  measures  are  organized 
around  die  accepted  military  analysis  factors  of  METT-T  (mission,  enemy  forces,  friendly  troops, 
terrain  and  time).  Since  measures  are  specific  to  particular  types  of  missions  and  time  is  a 
scenario-driven  constant  at  the  NTC.  this  portion  of  the  model  reduces  to  the  lollowing  three 
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indices  which  may  he  treated  separately  or  combined  to  form  a  composite  inilex:  1 )  percentage 
of  friendly  forces  (major  instrumented  weapons  systems  remaining  at  the  end  of  a  combat 
mission);  2)  percentage  of  enemy  forces  killed;  and  3)  terrain  control,  as  measured  by  the 
percentage  of  weapons  systems  crossing  a  defensive  boundary. 


TASK  PERFORMANCE 


UNIT  PERFORMANCE 


Figure  2,  Measurement  model. 
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Tlic  critical  task  performance  component  of  the  motle)  provides  an  immediate  process 
measure  of  unit  behaviors  that  contribute  to  mission  accomplishment  (Lewman,  in  preparation). 
Funhermore,  because  critical  tasks  are  doctrinaliy  based,  measures  of  task  performance  indicate  the 
degree  of  unit  conformity  to  tactical  doctrine. 

The  methodology  used  at  ARI-POM  to  identify  critical  tasks  has  several  important  features. 
First,  it  is  based  on  a  collective  front  end  analysis  which  drew  on  doctrinaliy  identified  tasks  as  a 
starting  point.  This  analysis,  performed  by  BDM  for  ARI-POM,  was  conducted  on  all  relevant 
doctrinal  manuals  at  Platoon,  Company,  and  Battalion  levels  available  in  the  Spring  of  1986 
Ihroiigh  ihc  Spring  of  \^81.  Tlie  sources  have  been  idenlified  in  Lewman  (In  preparation). 

These  tasks  were  carefully  analyzed  and  considered  for  their  criticality  by  military  subject  matter 
experts  at  the  NTC,  the  Combined  Arms  Training  Activity,  the  Infantry  and  Armor  Schools,  and 
selected  FORSCOM  units.  Thus,  task  lists  represent  a  culled  set  of  critical  tasks  substantially 
reduced  in  size  from  a  standard  ARTEP/AMTP  set  that  has  survived  considerable  scrutiny.  (See 
Figure  3.) 
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Figure  3.  Number  of  critical  NTC  tasks. 


Second,  the  tasks  have  been  organized  into  a  battle  flow  firework  which  identifies  battle 
phases  and  seginents  with  a  specific  purpose  and  observable  outcome  as  shown  in  Figure  4. 
Fuithermore,  critical  tasks  have  been  liriked  within  this  organizing  structure  to  show  sequential 
and  funcdonal  linkages.  (See  Figtm  5.)  Taken  together,  the  batde  flow  framework  and  the  task 
linkage  structures  provide  organizing  structures  for  data  collection  (either  manual  or  computer- 
aided  with  some  t]^  of  Electronic  Gipboard)  as  well  as  an  analytic  ftamework  for  examining  pat 
terns  of  task  performance. 


Figure  4.  Battle  flow  framework. 
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EXECUTION  PHASE  SEQUENCING 
PLATOON  HASTY  ATTACK 


End  Of  Mission 


MISSION  TASKS 


30 

React  to  Unexpected  Enemy  Contact 

41 

Respond  to  NBC  Operations 

31 

React  to  Change  in  Situatioa 

42 

Maintain  Communications 

32 

Conduct  Fue  and  Movement 

43 

Control  Fires 

33 

Conduct  Assault 

44 

Maintain  Lateral  Contact  with  Adjacent  Units 

34 

Acquire  and  Engage  Targets 

79 

React  to  Enemy  Air 

35 

Conduct  Consolidation 

92 

Control  Movement 

36 

Execute  Hre  Support  Plan 

103 

Move  to  Assault  Position 

37 

Support  Breaching  Effort 

111 

Mark  Breach 

38 

Conduct  Breach  of  Obstacle 

131 

Issue  FRAOO 

39 

40 

Reorganize  Assets 

Conduct  Evacuation  Procedures 

132 

Comply  with  Commander’s  Intent 

Figure  5.  Task  linkage  structure  for  execution  phase  of  platoon  hasty  attack. 
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The  third  type  of  measurement  specified  by  the  measurement  model  in  Figure  2  is  expert 
judgment  The  primary  need  for  expert  judgment  is  to  gain  creadve  insights  on  little  performance 
that  may  be  missed  by  mechanically  scored  outcomes  and  task  performance.  Expm  judgn^nt  on 
five  types  of  dimensions  have  been  identified  as  useful  indicators  in  assessing  unit  capabilities:  1) 
accomplishment  of  mission  segment/phase  outcomes;  2)  effectiveness  of  battlefield  of^ating 
systems;  3)  capability  on  key  performance  categories  (such  as  move,  shoot,  communicate);  4) 
overall  mission  effectiveness  rating;  and  S)  representativeness  of  mission  (or  identification  of 
unusual  circumstances). 

In  sum,  the  measurement  model  posits  three  forms  of  measurement  for  assessing  unit 
performance:  task  performance,  mission  outcomes,  and  expen  ratings  of  performance.  Depending 
on  the  purpose  for  which  data  are  being  used  (for  example,  diagnostic  feedback  or  systemic 
analysis),  these  measures  may  be  used  as  independent  performance  criteria  or  combined  to  form 
composite  measures  or  a  criterion  index.  However,  a  variety  of  cautions  must  be  considered 
when  using  such  measures  collected  in  a  field  environment  (see  Hiller,  1987  for  a  thcneiugh  discus¬ 
sion  of  these  issues). 


Methodology  for  SIMNET  Application 

In  developing  a  concept  for  a  common  performance  measurement  system,  it  is  important  to 
recognize  that  while  the  NTC  and  SIMNET  share  many  commonalities  they  also  have  many  unique 
features  from  a  performance  assessment  perspective.  Thus,  our  general  approach  has  been  to 
examine  the  applicability  of  the  NTC-ba^  model  to  SIMNET,  to  identify  the  boundaries  of 
overlap  in  the  two  settings,  and  to  examine  the  unique  measurement  capabilities  of  SIMNET  for 
each  ccMnponent  of  the  measurement  model.  Figure  6  presents  an  example  of  the  boundaries 
between  NTC  and  SIMNET  capabilities  and  the  unique  aspects  of  each  training  setting. 


NTC 

(e.g.,  defillade 
positioning, 
use  of  terrain 
features  for 
movement,  land 
navigation) 


(e.g..  AMTF 
Task  MOP'S, 
Position 
Location, 
Mission 
Outcomes 


SIMNET 

(e.g..  Target  Availability, 
Latency  of  Firing, 
Accuracy  of  Firing) 


Figure  6.  General  performance  measurement  approach. 
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As  noted  at  the  outset  of  this  report,  the  research  scope  for  this  effort  was  determined  based 
on  guidance  from  ARI  and  CATA  at  the  project’s  June  In-Progress  Review  (IPR)  a^ut  frequency 
and  type  of  training  missions  anticipated  in  initial  SIMNET  usage  for  urtit  training.  The  scope  for 
the  work  described  in  this  report  includes  two  missions:  platoon  defense  and  platoon  hasty  attack. 
After  the  scope  of  the  effort  was  identified,  the  next  step  was  to  determine  the  relevance  of  the 
critical  tasks  identified  in  the  previous  ARI  research  focused  on  the  NTC  to  SIMJ4ET.  Each  critical 
task  was  categorized  as:  1)  performed  in  the  SIMNET  device;  2)  performed  in  the  SIMNET 
complex;  3)  supported  in  the  SIMNET  complex  with  nnodifications  or  additions;  or  4)  not  sup¬ 
ported.  The  results  of  this  analysis  are  presented  in  the  following  section  of  this  report. 

For  all  critical  tasks  identified  as  potentially  supportable  in  the  SIMNET  device  or  complex, 
measures  of  performance  (MOPs)  based  on  the  previous  ARI  research  were  examined.  This 
analysis  suggested  that  most  of  the  previously  identified  MOPs  could  be  strengthened  by  further 
operationalization.  Therefore,  the  decision  was  made  to  use  the  performance  standards  specified  in 
the  previous  research  as  the  starting  point  for  generating  MOPs. 

Three  types  of  MOPs  have  been  generated  based  on  the  specific  requirements  of  the  critical 
task.  Automated  MOPs  arc  those  which  can  be  gathered  from  the  SEMNCT  data  stream.  We  have 
given  priority  to  these  measures  in  an  effort  to  increase  reliability  and  to  reduce  the  data  collection 
burden  on  observers.  Observable  MOPs  are  those  which  can  be  gathered  by  direct  observation  of 
task  performance,  indirect  observation  using  the  planview  display,  or  expert  judgment  based  on  a 
synthesis  of  the  automated  and  observable  MOPs.  Other/additional  MOPs  are  those  which  would 
require  either  a  change  in  the  SIMNET  facility  (such  as  the  acquisition  of  a  multi-channel  recender 
for  radio  nets)  or  a  tnodification  to  simulation  software.  There  are  a  number  of  impomnt  issues 
and  tradeoffs  which  emerged  from  this  activity  which  are  discussed  in  the  following  section  of  this 
report. 


All  MOPs  were  subjeaed  to  internal  review  for  their  technical  and  substantive  soundness 
and  revisions  made  where  necessary.  While  this  report  as  the  documentation  of  the  measurement 
concept,  there  are  several  research  steps  which  must  be  undertaken  to  translate  this  measurement 
concept  into  a  fully  functional  feedback  system  for  unit  performance.  These  requirements  are 
summarized  in  the  concluding  section  of  this  report. 


Key  Findings 

A  number  of  important  findings  and  critical  issues  emerged  in  the  process  of  developing  a 
common  performance  measurement  concept  for  NTC  and  SIMNET.  These  are  suirunarized  below 
with  the  discussion  organized  by  the  three  major  forms  of  measurement  inherent  in  the 
measurement  model:  mission  outcomes,  task  performance,  and  expen  judgment 

Mission  Outcomes 

As  noted  earlier  in  this  report,  the  mission  outcome  indices  derived  for  NTC  missions 
included  three  measures:  enemy  attrition,  friendly  attrition,  and  terrain  control.  In  considering  the 
applicability  of  these  measures  to  SIMNET,  it  is  important  to  take  into  account  that  missions 
conducted  at  the  NTC  are  generally  battalion  task  force  missions.  Battalions  are  inherently  capable 
of  operating  as  independent  units  (i.e.,  separate  from  Brigade)  because  of  their  size  and  resources. 
Thus,  terrain  control  (as  well  as  destruction  of  the  enemy  and  preservation  of  the  friendly  force)  are 
legitimate  purposes  of  a  battalion  task  force  mission.  Therefore,  the  three  measures  above 
represent  appropriate  measures  of  mission  effectiveness  at  the  battalion  task  force  level. 


However,  the  focus  of  the  present  analysis  is  on  two  platoon  missions:  defense  and  hasty 
attack.  Thus,  the  question  of  the  applicability  of  these  measures  not  only  to  the  SIMNET 
environment,  but  ^so  to  platoon  level  missions  generally,  must  be  addressed.  It  should  be  noted 
that  extension  of  this  methodology  to  other  Platoon  missions  would  require  a  specification  of  the 
doctrinal  purpose  of  those  missions.  It  is  the  purpose  of  the  mission  that  dictates  the  appropriate 
outcome  indices  to  be  used  to  measure  mission  p^ormance.  In  the  case  of  the  selected  two 
missions,  this  includes  the  three  indices  described  above.  For  other  missions,  e.g.,  road  march, 
these  indices  might  be  replaced  with  other  more  appropriate  indices. 

In  addressing  this  question,  it  is  important  to  note  that  platoons  (whether  at  NTC  or  within 
SIMNET)  always  operate  within  the  context  of  the  larger  company  and  battalion  mission.  This 
reality  has  two  important  implications  for  measuring  £e  effectiveness  of  platoon  missicms.  First, 
battalions  are  the  smallest  unit  given  terrain  control  responsibility  on  the  battlefield.  Platoons  (and 
companies)  do  not  have  the  resources  to  control  terrain;  thus,  platoons  use  fire  and  maneuver  to 
support  the  mission  of  higher  echelons.  Therefore,  only  measures  related  to  enemy  attrition 
Oethality)  and  friendly  forces  (survivability)  are  appropriate  indices  of  mission  effectiveness  for 
platoons.  These  measures  are  easily  gathered  from  the  automated  data  stream  supporting 
SIMNET. 

Second,  platoons  are  often  assigned  different  roles  in  the  support  of  the  mission  of  higher 
echelons,  particularly  in  the  attack.  For  example,  typically  platoons  are  assigned  one  of  four  roles 
in  a  battalion  attack;  a  breaching  force,  a  security  fnce,  an  assault  element,  or  an  attack  by  fire 
elenrenL  The  breaching  element  has  the  responsibility  to  neutralize  enemy  obstacles  and  to  provide 
security  to  the  breaching  fcwce;  the  assault  element  is  assigned  to  move  on  the  objective  whQe  the 
attack  by  fire  element  suppresses  enemy  fire.  The  implication  here  is  that  measures  of  enemy  and 
friendly  attrition  must  be  interpreted  relative  to  die  role  of  the  platoon.  Thus,  standards  for 
effective  levels  of  "lethality"  or  "survivability"  must  be  established  with  respect  to  the  role  of  the 
platoon. 

Figure  7  presents  standards  for  interpreting  the  effectiveness  of  platoons  in  the  defense. 
These  staiaiards  were  derived  based  on  inputs  from  subject  matter  expem  including 
observer/controllers  at  the  NTC  and  doctrine  writers  at  the  Armor  School.  (See  Lewman,  Root, 
2^mmerman,  and  Baldwin,  in  preparation,  for  a  fuller  discussion  of  the  methodology  for  deriving 
platoon  mission  standards.)  These  platoon  mission  standards  for  the  defense  are  equally  applicable 
in  the  NTC  and  SIMNET  environments.  In  fact,  as  SIMNET  data  become  available,  noimative 
distributions  of  performance  on  lethality  and  survivability  indices  can  be  derived  as  further  input 
into  the  standard  setting  process. 


Mission  Purpose: 

Destroy  Enemy  Force 

Mission  Performance  Standards 

Enemy  Forces 

Destroy  70%  of  OPFOR 

(Lethality) 

in  Sector 

Friendly  Forces 
(Survivability) 

Retain  2  Combat  Systems 

Figure  7.  Mission  effectiveness  standards  for  platoon  defense. 
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As  shown  in  Figure  8  below,  the  platoon  attack  mission  is  somewhat  more  complex. 
Platoons  may  be  assigned  one  of  four  responsibilities  as  identified  at  the  top  of  the  figure. 
Standards  for  lethality  and  survivability  derived  from  subject  matter  experts  (Lewman  et  al.,  in 
preparation)  take  into  account  the  responsibility  of  the  platoon  in  the  attack.  The  most  problematic 
area  for  standard  setting  emerging  from  this  research  involves  the  suppression  of  enemy  firing 
activity.  The  question  of  enemy  firing  levels  an^or  patterns  that  constitute  effective  suppression  is 


Mission  Purpose  Breach  Enemy  Obstacles 


Provide  Security 

Assault  the  Objective 

Suppress  Enemy  Fire 

Mission  Performance  Standards 

Breaching  Element* 

--  Breach  Force 

Enemy  Forces 

Neutralize  Obstacle 

(Lethality) 

No  Fratricides  Due  to  Obstacle 

Friendly  Forces 
(Survivability) 

Retain  60%  of  Force 

••  Security  /  Suppression  Force 

Enemy  Forces 

No  Losses  to  Breach  Force  as  a  Result 

(Lethality) 

of  OPFOR  Direct  Rre 

Friendly  Forces 
(Survivability) 

Retain  2  Combat  Systems 

Assault  Element 

Enemy  Forces 
(Lethality) 

Suppressed  Firing  Activity  (TBD) 

Friendly  Forces 
(Survivability) 

Retain  2  Combat  Systems 

Attack  by  Rre  (Suppression) 

Enemy  Forces 
(Lethality) 

Suppressed  Firing  Activity  (TBD) 

Friendly  Forces 
(Survivability) 

Retain  2  Combat  Systems 

*  Not  currently  supported  in  SIMNET 


Figure  8.  Mission  effectiveness  standards  for  platoon  hasty  attack. 
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very  complex  and  is  cunently  under  further  investigation  in  ARI  sponsored  research.  Indeed, 
SIMNET  provides  a  control!^  environment  for  examining  such  issues  and  data  from  SIMNET 
battles  may  provide  important  insights  into  such  questions  as  data  become  available.  More 
generally,  while  the  standards  fw  the  assault  and  attack  by  fire  elements  are  applicable  in  the 
SIMNET  environment,  obstacle  breaching  is  not  presently  supported. 

Several  cautions  must  be  raised  about  the  interpretation  and  use  of  mission  effectiveness 
indices  such  as  friendly  attrition  and  enemy  attridtm.  The  first  concerns  the  application  of 
standards  generally  to  mission  perfcnmance  criteria.  While  it  is  common  practice  in  Army  training 
to  establish  standards  as  cutoff  criteria  forjudging  task  performance,  the  concept  of  standards 
applied  to  mission  effectiveness  is  a  potentially  hazardous  undertaking.  First,  it  must  be  under- 
st(^  that  there  are  no  currently  accepted  standards  within  the  Army  community  for  mission 
effectiveness  indices  such  as  lethality  and  survivability.  The  standards  laid  out  in  Figims  7  and  8 
were  derived  from  inputs  of  subject  matter  experts  (SMEs).  These  SMEs  were  primarily 
Observer/Controllers  at  the  NTC  who  have  wimess^  large  numbers  of  battles  and  who  used  their 
experience  to  develop  standards  on  the  mission  effectiveness  indices  which  in  their  experience 
reflected  a  breakpoint  forjudging  combat  effective  versus  ineffective  platoons.  However,  there  is 
considerable  difference  of  opinion  in  Army  circles  about  these  types  of  standards  and  further 
research  is  needed. 

A  second  caution  concerns  the  role  that  conditions  play  in  mission  performance.  Battlefield 
conditions,  particularly  the  speciflc  terrain  on  which  missions  are  executed,  and  the  force  levels  of 
the  opposing  sides,  are  likely  to  influence  the  difficulty  of  the  mission,  and  place  boundary  limits 
on  effectiveness.  At  present,  these  influences  are  not  well  understood  and  research  is  ongoing  to 
examine  the  influence  of  conditions  (especially  terrain)  on  mission  complexity  and  ultimately 
mission  accomplishment  Thus,  care  must  be  exercis(^  in  interpreting  measures  of  mission 
effectiveness  under  differing  sets  of  conditions. 

Finally  and  perhaps  most  importantly,  the  use  to  which  mission  outcome  assessments  are 
put  must  be  carefully  examined.  Such  measures,  either  treated  as  separate  indices  or  axnbined  into 
a  suinmary  index,  provide  a  gauge  for  the  validity  of  tactical  doctrine.  Examining  the  effectiveness 
of  missions  execut^  in  accordance  with  tactical  doctrine  on  repeated  occasions  provides  a  tool  for 
demonstrating  the  value  of  current  doctrine  and  for  identifying  areas  of  weakness.  Furthermore, 
by  examining  the  effectiveness  of  missions  executed  in  innovative  or  non-doctrinal  ways,  lessons 
learned  on  doctrinal  improvements  can  be  derived.  Such  uses  of  outcome  data  in  the  aggregate 
(with  protection  of  unit  identification  and  proper  database  security)  appear  to  be  well  acc^t^ 
within  the  Army  community.  However,  the  use  of  outcome  assessments  as  part  of  the  diagnostic 
feedback  process  to  units  is  somewhat  more  sensitive.  Most  Army  bainers  would  agree  that 
survivability  and  lethality  indices  provide  a  useful  context  for  examining  unit  perfcxmance  and 
identifying  training  strengths  and  weaknesses.  However,  this  agreement  disintegrates  when  such 
indices  are  intend^  to  be  used  to  "grade"  units  and  to  compare  units  against  each  other.  Such 
cross-unit  comparisons  are  fraught  with  problems  given  current  limitations  in  our  understandings 
of  battlefield  dynamics  and  the  influence  of  varying  conditions. 

Critical  Task  Performance 

A  second  major  component  of  the  measurement  nxxlel  centers  on  unit  performance  of 
mission  critical  tasks.  Two  important  aspects  of  our  analysis  of  critical  task  performance  within 
the  context  of  NTC  and  SIMNET  missions  merit  discussion.  The  first  focuses  on  the  applicability 
of  critical  tasks  in  the  two  settings,  the  second  on  the  development  of  measures  of  performance 
(MOPs)  for  common  critical  tasks.  Key  fmdings  and  issues  that  must  be  address^  are  discussed 
below  for  each  of  these  areas. 


12 


Applicability  of  Critical  Tasks 


Prior  to  analyzing  the  applicability  of  NTC-dcrivcd  tasks  to  SIMNET,  we  examined  the 
overlap  on  NTC-derived  tasks  across  missions.  The  purpose  of  this  preliminary  analysis  was  to 
determine  the  impact  of  the  initial  research  scope  of  the  range  of  critical  tasks  to  be  considered. 

The  results  of  this  overlap  analysis  at  the  platoon  level  can  be  found  in  Figures  1  through  3  and 
Table  A-1  of  Appendix  A.  To  summarize  these  results,  selection  of  the  two  Platoon  missions 
includes  37  of  the  51  (72.5%)  planning  critical  tasks,  23  of  the  28  (82.1%)  of  the  preparation 
critical  tasks,  and  30  of  the  36  (83.3%)  execution  critical  tasks  in  at  least  one  of  the  missions. 

The  results  of  this  analysis  confirmed  our  expectation  that  the  selection  of  the  defend  and 
hasty  attack  missions  as  a  starting  point  would  provide  a  broad  platform  of  critical  tasks  from 
which  to  work.  In  all  three  phases,  less  than  20%  of  the  platoon  tasks  for  that  phase  must  be 
performed  in  both  defense  and  hasty  attack.  Thus,  these  two  missions  provide  a  diverse  set  of 
platoon  tasks  for  analysis. 

The  results  of  our  analysis  of  the  applicability  of  mission  critical  tasks  derived  at  the  NTC 
to  SIMNET  are  summarized  in  Figure  9  for  the  Platoon  Defend  mission  and  in  Figure  10  for  the 
Platoon  Hasty  Attack  mission.  Squares  represent  tasks  that  are  currently  suppon^  in  the 
SIMNET  device  (unshaded)  or  in  the  SIMNET  complex  (shaded).  Circles  represent  tasks  that 
either  could  be  supported  with  modifications  or  additions  to  the  SIMNET  complex  [such  as  by 
reconfiguration  of  space  or  acquisition  of  new  equipment]  (unshaded)  or  are  not  presently 
supported  (shaded). 

Several  key  findings  emerged  from  these  analyses.  First,  a  large  number  of  NTC  critical 
tasks  can  be  trained  on  SIMNET.  More  specifically,  roost  planning  and  many  preparation  tasks 
can  be  performed  in  the  SIMNET  complex.  Furthermore,  Ac  majority  of  execution  tasks  can  be 
performed  in  Ae  SIMNET  device.  Table  1  provides  Ae  number  and  percent  of  critical  tasks  by 
phase  for  Ae  two  missions  Aat  are  supportable:  in  Ae  SIMNET  device;  in  Ae  SIMNET  complex; 
in  the  SIMNET  complex  with  modifications;  or  not  supported.  These  results  show  Aat  a  very  high 
percentage  of  the  tasks  are  supported  by  SIMNET  across  phases  and  missions. 


MISSION: 

PLATOON 

DEFEND 

SIMNET 

SIMNET 

COMPLEX 

NOT 

DEVICE 

COMPLEX 

\WrTHMOD. 

SUPPORTED 

PLAN  (N-24) 

1  (4.2) 

22  (91.6) 

1  (4.2) 

0 

(0.0) 

PREPARE  {N-19) 

2  (10.5) 

7  (36.8) 

5  (26.3) 

5 

(26.3 

EXECUTE  (N-18) 

15  (83.3) 

0  (0.0) 

1  (5.5) 

2 

(11.1) 

MISSION: 

HASTY  ATTACK 

SIMNET 

SIK«4ET 

COMPLEX 

NOT 

DEVICE 

COMPLEX 

WITHMOD. 

SUPPORTED 

PLAN  (N-22) 

0  (0.0) 

22  (100) 

0  (0.0) 

0 

(0.0) 

PREPARE  (N-13) 

1  (7.7) 

8  (61.5) 

3  (23.1) 

1 

(7.7) 

EXECUTE  (N-21) 

16  (83.3) 

0  (0.0) 

1  (4.7) 

4 

(19.1) 

NOTE;  Tha  5  common  tasks  in  aach  mission  ara  supportaUa  in  tha  SIMNET  complax  during  Planning  and 

Prsparation,  and  in  tha  SIMNET  davica  during  Exaculion.  Numbars  in  paranthasas  ara  parcantaga  of  critical 
tasks  for  tha  phasa. 

Table  1.  Applicability  analysis  for  platoon  missions. 
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DEFEND  •  MISSION  TASKS 


^rN.r^r*->r»r^r^r»r^r^r^S 
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Figure  9.  Mission  Task  List  for  Platoon  Defend. 


Platoon  -  Defend 


Figure  9a-  Applicability  of  Critical  Task  Linkage  Structure  for  Platoon  Defend. 
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HASTY  ATTACK  -  MISSION  TASKS 
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Figure  10.  Mission  Task  List  for  Platoon  Hasty  Attack. 


fLANNil 


Platoon  Hasty  Attack 


Figure  10a.  Applicability  of  Critical  Task  Linkage  Structure  for  Platoon  Hasty  Attack. 
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However,  there  are  some  significant  limitations  in  task  performance.  First,  our  analysis 
indicated  that  some  tasks  can  only  partially  trained  generally  because  some  subtasks  are  not  sup- 
ported  in  the  SIMNET  environment.  For  example.  Task  108  [Maintain  Operations  Security]  for 
Platoon  Defense  has  four  subtasks:  1)  maintain  signal  security;  2)  maintain  information  security;  3) 
conduct  counterintelligence  measures;  and  4)  maintain  physic^  security.  Only  the  first  subtask  can 
be  fully  trained  within  the  SIMNET  environment  using  tactical  radio/telephone  communication 
between  simulators  or  areas  of  the  SIMNET  complex.  In  contrast,  the  second  subtask  cannot  be 
trained  at  all  because  it  requires  use  of  physical  terrain  and  equipment  that  are  currently  unavailable. 
Further,  only  aspects  of  the  third  and  founh  subtasks  can  be  practiced  in  the  SIMNET 
environment. 

Second,  some  tasks  can  be  planned  or  prepared  for  in  the  SIMNET  complex  (if  desired); 
however,  they  cannot  be  executed  in  the  SIMNET  device  at  present  These  tasks  primarily  involve 
NBC  (nuclear,  biological,  chemical)  operations,  obstacle  emplacement  and  breaching,  casualty 
evacuation  and  the  use  of  dismounted  infantry. 

Third,  the  training  value  for  some  tasks  is  limited  by  the  capability  of  the  simulation  to 
replicate  terrain  usage.  These  limitations  are  particularly  evident  in  the  defense  where  vehicle  usage 
of  cover  and  concealment  and  detilade  positions  are  partially  supponed,  but  the  difficulty  of  the 
task  and  the  interaction  with  the  terrain  is  not  consistent  with  performance  of  the  same  task  in  the 
field. 


Some  of  the  current  limitations  in  training  tasks  on  SIMNET  will  be  overcome  when  the 
production  version  of  SIMNET  (CCTT)  is  fleld^.  Others  could  be  addressed  through  additions  to 
the  SIMNET  complex  or  software  modifications.  Others  may  be  best  dealt  with  through  the 
development  of  an  explicit  training  strategy  which  identifies  the  role  of  SIMNET  training  in 
conjunction  with  other  unit  training.  The  point  here  is  that  a  training  strategy  should  provide  an 
integrated  approach  for  ensuring  that  units  have  the  opportunity  to  train  on  the  full  range  of  mission 
critical  tasks,  not  that  any  one  training  ^proach  such  as  SIMh^T  should  necessarily  train  fuUy  on 
all  tasks.  The  Device-Based  Field-Supported  training  strategy  articulated  by  Beecher  (1988)  is  one 
example  of  such  an  integrated  training  approach. 


One  final  point  about  the  perspective  adop^  in  this  wplicability  analysis  deserves 
mention.  We  have  examined  the  applicability  of  training  NT(J-derived  critical  tasks  in  SIMNET 
with  a  view  of  SIMNET  as  primarily  a  tactical  maneuver/command  and  control  trainer.  Experience 
at  the  NTC  has  shown  that  tasks  critical  to  mission  success  are  collective  (primarily  maneuver)  and 
leadership  (mainly  command  and  control)  tasks.  Thus,  the  NTC-derived  tasks  tend  to  be  higher 
order  group  tasks  or  cognitively  oriented  leader  tasks.  We  have  examined  the  capability  of 
SIMNET  to  exercise  the  performance  of  these  tasks,  not  necessarily  the  c^abili^  of  SIMNET  to 
replicate  the  performance  of  these  tasks  exactiy.  I^m  our  perspective,  selective  fidelity  represents 
a  continuum  with  more  or  less  fidelity  required  depending  on  the  nature  of  the  skills  to  be  trained 
and  the  place  of  the  training  device  in  the  overall  unit  training  program.  In  contrast,  Drucker, 
Campshure,  and  Campbell  (1988)  conducted  a  similar  analysis  of  SIMNETs  capabilities  to 
provide  a  simulated  environment  for  crew  operations  and  individual  crew  meml^r  jobs.  This 
analysis  had  a  irx>re  concrete  focus  on  individual  skills  and  the  operational  mechanics  of 


performing  mecinc  jobs  and  identified  many  shortcomings  of  SIMNET  ui  allowmg  performance 
or  practice  of  these  types  of  tasks.  These  shortcomings  primarily  revolved  around  selective  fidelity 
issues.  We  do  not  regard  their  findings  as  contradictory  to  ours;  rather  we  see  the  differences  as 
emerging  from  varying  perspectives  and  perhaps  different  assumptions  about  the  role  of  SIMNET 
within  the  context  of  a  larger  unit  training  program. 


1 
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Development  of  Measures  of  Performance  (MQPs^ 


The  second  major  component  of  our  analysis  of  critical  task  performance  centered  on  the 
development  of  measures  of  performance  (MOPs)  for  common  cridcal  tasks.  As  noted  earlier, 
these  MOPs  were  derived  from  the  performance  standards  for  tasks  and  subtasks  that  survived  the 
applicability  analysis.  The  measurement  concept  for  three  platoon  missions  can  be  found  in 
Kerins  and  Atwood  (in  preparation  a,  b,  and  c)  and  for  the  corresponding  company  missions  in 
Kerins,  Root  and  Atwo^  0989a,  1989b,  and  1989c) . 

The  measurement  concept  documented  in  each  of  these  reports  is  ordered  by  battle  phase 
(i.e.,  plan,  prepare,  execute)  and  by  se^ent  within  phase  (e.g.,  conception  segment  of  the 
planing  phase).  For  each  common  critical  task,  the  task  and  any  component  subtasks  are  identi¬ 
fied  along  with  the  doctrinal  standard  and  reference.  (Dn  the  companion  page,  performance 
standards  for  the  task  or  component  subtasks  are  presented  in  conjunction  with  MOPs. 

A  number  of  clarifying  points  about  the  MOPs  presented  in  these  documents  warrant 
discussion.  First,  MOPs  specify  variables  which  serve  as  indices  of  performance.  That  is,  they 
provide  specifications  of  the  perftxmance  dimensions  which  can  be  used  to  assess  whether  a  unit 
has  met  the  performance  standard.  As  such,  MOP  specification  is  part  of  the  design  phase  of  a  unit 
performance  measurement  system.  In  order  to  build  a  fully  functional  feedback  system,  tiiese 
MOPs  must  be  translated  into  collection  tools  aiKl  strategies  developed  for  analysis,  interpretation 
and  presentation.  Aspects  of  these  activities  are  part  of  other  subtasks  in  this  research  effort  and 
are  discussed  further  in  the  concluding  section  of  this  report 

Three  types  of  MOPs  have  been  specified  here.  Automated  MOPs  are  variables  which  can 
be  extraaed  from  the  autonuted  data  stream  driving  the  ^ulation  and  incwporated  into  an 
automated  database.  Observable  MOPs  are  performance  dimensions  which  may  be  observed  or 
assessed  by  a  human  observer  (e.g.,  dedicate  Observer/Controllers  or  designated  unit  personnel). 
Other/additional  MOPs  are  those  wMch  could  be  gathered  through  a  change  to  the  SIMNFT  facility 
(such  as  the  acquisition  of  equipment)  or  through  a  modification  to  the  SIMNBT  software. 

There  are  a  number  of  issues  that  must  be  recognized  in  examining  the  automated  and 
observable  MOPs  and  tradeoffs  that  must  be  weighed.  First,  we  have  given  priority  to  the 
specification  of  automated  MOPs  in  accordance  with  guidan^  fiom  AM  and  our  undentandings  of 
current  plans  for  fielding  and  staffing  SIMNET  at  home  stations.  In  some  cases,  these  automated 
indices  provide  a  direct  and  uiuunbi^ous  measure  of  performance  of  a  task  or  subtask;  however, 
in  other  cases  they  require  combination  and  interpretation. 

For  example.  Task  76  from  the  execution  phase  of  the  Platoon  Defend  mission  is  to 
"Execute  battle  handover".  There  are  three  performance  standards  associated  with  this  task.  The 
first  is  "there  are  no  friendly  casualties  as  a  result  of  friendly  fires".  In  this  case,  the  automated 
MOP  is  straightfnward:  the  number  of  firatricides  (friendly  vehicles  Idlled  or  tut  by  the  defending 
unit).  In  contrast,  the  second  performance  standard  is  "platoon  maintains  continuous  effective  fires 
on  advancing  enemy  elements".  There  a  number  of  automated  MOPs  that  bear  on  this  performance 
standard  iiKluding  die  number  and  type  of  rounds  fired,  the  number  and  location  of  enemy  vehicles 
hit  or  killed  and  the  number  and  type  of  indirect  fire  missions.  However,  these  are  indirect 
indicators  which  must  be  combing  and  interpreted  by  a  subject  matter  expert  to  arrive  at  a 
judgment  about  whether  (or  the  degree  to  which)  platoon  files  were  continuous  and  effective  under 
those  batde  conditions. 


Furthermore,  the  automated  MOI^  are  limited  in  number  and  substantive  focus  (generally 
to  movement,  position  location,  and  firing  activity)  because  of  the  nature  of  the  data  stream 
supporting  SIMNET.  Many  of  these  automated  measures  can  serve  as  indirect  indicators  of  the 
pmormance  of  the  majority  of  execution  tasks.  From  a  measurement  perspective,  this  overlap 
leads  to  nonindependence  of  measures  and  considerable  covariation  of  performance  indicators 
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across  tasks.  While  this  problem  is  somewhat  attenuated  by  the  fact  that  indicators  are  likely  to  be 
combined  and  interpreted  in  somewhat  different  ways  given  the  nature  of  the  task  to  which  they  are 
applied,  the  issue  r^uires  consideration  of  technicd  implications  as  well  as  other  measurement 
alternatives  beyond  ^e  automated  data  streant 

Second,  the  observable  MOPs  bring  other  issues  to  bear.  Measures  of  performance  of 
many  tasks,  especially  those  in  the  planning  and  preparation  phases,  can  be  physically  observed  in 
the  SIMNET  complex.  In  scxne  cases,  these  indicatcvs  are  fairly  direct  and  straightforward 
measures  which  could  be  gathered  fairly  reliably  by  an  observer  with  some  training.  However,  in 
other  cases,  the  indicators  must  be  inferred  from  behavior  because  the  ta^  involves 
mental/cognitive  activity. 

For  example.  Task  127  in  the  planning  phase  of  the  Platoon  Defense  mission  is  to 
"Conduct  mission  analysis".  This  task  is  largely  a  cognitive  analytic  activity  that  must  be 
accomplished  by  the  platotm  leader  which  is  not  directly  observable.  The  jx^ormance  standards 
for  this  task  relate  to  complete  identification  of  specified  and  implied  tasks  and  accurate  mission 
analysis  on  the  factors  of  METT-T.  Thus,  the  r^server  would  need  to  watch  for  a  variety  of 
behavioral  manifestations  (such  as  study  of  the  order  from  higher,  discussions  with  staff  or  talking 
aloud)  that  would  allow  turn  to  formulate  a  judgment  on  the  completeness  artd  accuracy  of  the 
mission  analysis.  In  contrast.  Task  128  in  the  planning  phase  is  to  "Initiate  the  planning  process". 
This  task  primarily  involves  using  the  reverse  planning  process  to  allocate  time,  assigning  specific 
responsibilities  to  platoon  members,  and  issuing  preliminary  graphics.  These  indicators  can  be 
dir^tly  observed  and  assessed  using  tangible  evidence  by  an  observer  in  a  much  more 
straightforward  way.  Clearly,  the  training  demands  on  observers  to  make  these  two  types  of 
judgments  reliably  vary  gready. 

Further,  performance  of  some  tasks  cannot  be  observed  direcdy  (because  of  the  closed  box 
simulator)  but  must  be  inferred  based  on  indirect  indicators.  Fo^  example,  while  an  observer 
cannot  view  tank  crews  direcdy,  he  can  observe  their  location  and  activity  on  the  batdefield  using 
the  planview  display.  In  addition,  by  monitoring  the  communications  nets,  an  observer  can  gather 
additional  information  on  which  to  base  inferences  about  task  perfcxmance. 

In  sum,  there  are  number  of  tradeoffs  that  must  be  considered  in  determining  MOPs  that 
will  be  translated  into  collection  tools.  Automated  MOPs  have  the  advantage  of  easy  collection  but 
often  require  combination  and  interpretation  in  order  to  use  them  as  measures  of  ta^  performance. 
In  addition,  they  are  constrained  in  number  and  scope  leading  to  problems  of  overlap  and  nonin¬ 
dependence.  Observable  MOPs  are  a  prerequisite  for  assessing  many  types  of  tasks,  especially  in 
the  planning  and  preparation  phases.  However,  they  bring  demands  for  personnel  (human 
observers)  who  must  be  train^  to  make  reliable  judgments.  These  training  demands  vary  greatly 
depending  on  the  amount  of  inference  and  interpretation  required.  Furthermore,  MOPs  could  be 
considerably  strengthened  through  the  acquisition  of  new  equipment  in  the  SIMNET  complex 
(especially  a  multi-chaimel  recover  for  communications  nets)  and  throu^  modifications  to 
SIMNET  software.  However,  these  additions  and  modifications  also  carry  resource  implications. 
Thus,  resource  allocation  decisions  must  be  traded  off  in  light  of  the  technical  benefits  to  be 
gained.  We  have  purposely  laid  out  the  alternatives  here;  it  is  the  prerogative  of  the  government  to 
make  such  resourcing  and  policy  decisions. 

Expert  Judgment 

The  third  majcn- component  of  the  measurement  model  is  expert  judgment.  As  noted 
earlier,  the  primary  need  for  expert  judgment  is  to  gain  creative  insights  on  battle  performance  that 
may  be  missed  by  mechanically  scored  outcomes  and  task  performance. 
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Our  analysis  suggests  that  expen  judgment  on  all  five  types  of  dimensions  identified  as 
useful  indicators  at  the  OTC  can  serve  a  comparable  function  within  SIMNET.  These  dimensions 
include:  1)  accomplishment  of  mission  segment^hase  outcomes;  2)  effectiveness  of  battlefield 
operating  systems;  3)  capabili^  on  key  performance  categories  (such  as  shoot,  move,  and 
communicate);  4)  overall  mission  effectiveness  rating;  and  5)  representativeness  of  mission  (or 
idenaficadon  of  unusual  circumstances). 

It  is  important  that  when  these  dimensions  are  translated  into  a  collection  guide  for 
observers  that  a  requirement  be  levied  for  observers  to  supply  evidence  to  support  extremely  high 
or  low  numerical  ratings.  This  evidence  may  take  the  form  of  an  observation,  an  interview,  a 
planview  screen  display,  a  clip  of  a  communications  net,  or  a  unit  produced  document  such  as  an 
Operations  Oder.  Such  ratings  in  combination  with  tangible  evidence  provide  a  global  context  for 
interpreting  outcome  or  task-tniented  measurements  and  for  identifying  unique  sets  of 
circumstances  influencing  a  unit's  performance.  They  also  serve  as  potentiaUy  persuasive  tools  for 
use  in  an  After  Action  Review  cr-  other  diagnostic  fe^back  setting. 

Discussion 

The  concept  presented  in  this  report  has  several  important  implications  for  the  effective  use 
of  SIMNET  as  part  of  the  larger  Army  training  system.  First,  the  concept  operationalizes  the 
specific  tasks  and  measures  of  performance  within  die  confines  of  a  thr^-part  systems  model. 
Further,  this  model  has  been  applied  to  the  National  Training  Center  (NTC)  to  enhance  the  perfor¬ 
mance  measurement  and  feedback  at  this  advanced  training  »cility.  The  applicadtm  of  the  same 
model  and  attendant  methodology  helps  to  ensure  comparability  in  performance  measurement  and 
feedback  between  the  NTC  and  SIMI^T. 

As  with  the  application  of  this  measurement  model  and  approach  to  the  NTC,  several 
benefits  accrue  to  SIMNET.  First,  the  structures  inherent  in  the  model  provide  an  organizing 
structure  for  training  feedback.  In  particular,  the  Battle  Flow  Framework  and  Task  Linkage 
structures  provide  simple  graphical  tools  for  tracing  performance  and  identifying  major 
performance  breakdowns.  Thus,  these  structures  can  serve  to  guide  the  discussion  of  performance 
during  an  After  Action  Review  (AAR)  or  to  organize  textual  fe^back  for  a  Take  Home  Package 
(THP)  or  other  paper-based  report  Finally,  these  structures  make  it  simple  for  a  unit  commander 
or  other  relevant  ^rsonnel  to  quickly  spot  breakdowns,  patterns  of  performance,  and  strengths 
and  weaknesses  that  need  to  be  address^  in  subsequent  training. 

Beyond  the  enhanced  value  of  feedback  for  the  unit  the  concept  presented  here  offers  the 
potential  for  systemic  feedback  as  well.  The  approach  embodied  here  is  consistent  with  that 
employed  by  ^e  Center  for  Army  Lessons  Learned  programs.  Thus,  the  training  data  captured 
throu^  this  measurement  approach  can  provide  a  platform  for  generation  of  andyses  addressing 
the  areas  of  Doctrine,  Training,  Organization,  Materiel,  and  Le^ership  (DTOML).  Further,  the 
data  can  be  used  in  conjunction  wi£  comparable  data  ^m  the  NTC  to  broaden  the  scope  and 
increase  the  richness  of  these  analyses.  L^tly,  this  system  and  the  data  captured  by  it  provides  a 
tremendous  research  resource  for  use  by  the  Army's  research  agency,  ARI. 

An  important  issue  associated  with  the  ccRicept  presented  here  concerns  the  advisability  and 
applicability  of  the  end-of-mission  standards.  Recently,  senior  Army  leadership  has  revisited  the 
issue  of  applying  set  numerical  standards  to  mission  performance.  Senior  leaders  determined  that 
specific  numbers  could  not  be  set  for  the  many  conditions  under  which  the  missions  might  be 
performed.  Therefore,  the  current  guidance  is  that  it  is  the  commandei's  responsibili^  to 
determine  whether  mission  performance  met  the  standard  or  not  The  standards  provided  in  this 
document  are  not  viewed  as  contradictory  to  this  guidance.  Rather,  they  are  provided  as  a 
benchmark  derived  from  empirical  analyses  of  performance  at  the  National  Training  Center, 
modified  by  the  expen  judginent  of  the  Observer/(3ontroller  cadre  at  the  NTC.  We  are  not 
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recommending  the  use  of  these  standards  by  the  units  training  in  SIMNET  nor  their  inclusion  in  the 
unit's  AAR.  ^ther,  we  offer  them  as  one  source  of  data  that  might  be  considered  in  examining 
unit  performance,  or  more  appropriately,  to  be  used  by  researchers  who  might  need  to  derive 
higher  order  variables  to  summarize  unit  performance.  It  should  aiso  be  noted  that  while 
"standards"  were  provided  for  only  the  Platoon  Defense  and  Hasty  Attack  missions,  that  simple 
direct  extensions  of  these  would  provide  comparable  baselines  for  other  Platoon  or  Company 
missions.  In  sum,  we  felt  that  the  provision  of  this  information  was  a  necessary  component  of  a 
comprehensive  measurement  model  and  concept  fcv  SIMNET. 

An  ancillary  benefit  of  the  applicability  analysis  conducted  (m  each  of  the  mission  critical 
tasks  was  the  identification  of  modifications  or  additions  to  SIMNET  or  the  SIMNET  con:^)lex  that 
would  allow  for  performance  of  certain  tasks.  Generally,  these  additions  or  modifications 
involved  minor  software  additions  or  acquisition  of  new  equipment  for  use  in  the  ctxnplex.  The 
details  of  these  changes  have  been  provided  in  Appendix  B  of  this  report. 

The  measurement  concept  must  now  be  elaborated  through  a  systematic  design  process  in 
order  to  build  a  fully  functional  feedback  system.  This  process  includes  designing  collection  tools 
including  screen  displays  for  automated  data  and  collection  guides  for  observers.  In  addition, 
plans  for  data  ancJysis,  approaches  for  data  interpretation,  and  strategies  for  presentation  must  also 
be  formulated.  These  activities  constitute  next  steps  in  the  research  process.  Aspects  of  these 
steps  are  the  focus  of  subsequent  tasks  and  subtasks  in  this  research  effort 

Summary 

This  report  has  presented  a  concept  for  the  design  of  a  common  performance  measurement 
system  for  unit  training  at  the  National  Training  Center  (NTC)  and  with  Simulation  Networking 
(SIMNET).  The  report  uses  two  platoon  missions.  Defend  and  Hasty  Attack,  as  the  vehicle  for 
presenting  this  measurement  concept  The  concept  has  been  extended  to  Platoon  Movement  to 
Contact  mission  and  to  the  Company  level. 
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APPENDIX  A 
OVERLAP  ANALYSIS 
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Figure  A-1.  Overlap  of  platoon  planning  tasks  by  mission 
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Figure  A-2.  Overlap  of  platoon  preparation  tasks  by  mission 
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Figure  A*3.  Overlap  of  platoon  execution  tasks  by  mission 
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PLAN  (TOTAL  TASKS  ■  51) 


Defend 

Day  Attack 
Night  Attack 
MTC 

Hasty  Attack 

Day 

Defend  Attack 

17+  (33.3)++  5  (9.8) 

1  (2.0) 

Night 

Attack 

5  (9.8) 

21  (41.2) 

3  (5.9) 

MTC 

5  (9.8) 
17  (33.3) 
16  (31.4) 
2  (3.9) 

PREPARATION  (TOTAL  TASKS  «  28) 

Defend 

Day  Attack 
Night  Attack 
MTC 

Hasty  Attack 

Defend 

12  (42.9) 

Day 

Attack 

5  (17.9) 

0  (0.0) 

Night 

Attack 

5  (17.9) 

10  (35.7) 

0  (0.0) 

MTC 

5  (17.9) 
10  (35.7) 
10  (35.7) 
1  (35.7) 

EXECUTION  (TOTAL 

TASKS  «  36) 

Defend 

Day  Attack 
Night  Attack 
MTC 

Hasty  Attack 

Defend 

11  (30.6) 

Day 

Attack 

5  (13.9) 

0  (0.0) 

Night 

Attack 

5  (13.9) 

19  (52.8) 

0  (0.0) 

MTC 

5  (13.9) 
12  (33.3) 
12  (33.3) 
5  (13.9) 

•»01  agonal  entries  present  number  and  percent  of  tasks  unique  to 
mission. 

♦•^Numbers  In  parentheses  are  percent  of  critical  tasks  for  phase. 
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Attack 

5  (9.8) 
14  (27.5) 
13  (25.5) 
13  (25.5) 

6  (11.8) 


Hasty 

Attack 

5  (17.9) 

6  (21.4) 
6  (21.4) 
6  (21.4) 
5  (35.7) 


Hasty 
Attack 
5  (13.9) 

18  (50.0) 

18  (50.0) 

11  (30.6) 

1  (2.8) 

a  particular 


Table  A-1. 


Number  and  percent  of  critical  tasks  common  to  mission  pairs 
by  mission  phase 


APPENDIX  B 

RECOMHENOATIONS  FOR  ADDITIONS  OR  MODIFICATIONS 


Supported  in  CoroTex  With  Additions  or  Modifications 

Suggested  or  required  changes  to  the  SIHNET  facility  to  Improve 
the  quality  of  training,  the  training  experience,  and  the  training 
feedback . 

•  Communications 

Monitoring  capability 
Recording  capability 
Time  tagging 

Synchronized  with  actions  on  the  simulator 
Playback  for  AAR  purposes 

•  Minor  Software  Fixes 

Plan  view  display 

Graphic  control  measures 
Plans/operatlons  (unit) 

Fire  plans 
Sector  sketches 
Obstacle/barrier  locations 

•  Unit  Provided  Materials 


SOPS 
Orders 
Overl ays 


